To facilitate research on text generation, this paper presents a comprehensive and unified library, TextBox 2.0, focusing on the use of pre-trained language models (PLMs). To be comprehensive, our library covers $13$ common text generation tasks and their corresponding $83$ datasets and further incorporates $45$ PLMs covering general, translation, Chinese, dialogue, controllable, distilled, prompting, and lightweight PLMs. We also implement $4$ efficient training strategies and provide $4$ generation objectives for pre-training new PLMs from scratch. To be unified, we design the interfaces to support the entire research pipeline (from data loading to training and evaluation), ensuring that each step can be fulfilled in a unified way. Despite the rich functionality, it is easy to use our library, either through the friendly Python API or command line. To validate the effectiveness of our library, we conduct extensive experiments and exemplify four types of research scenarios. The project is released at the link: https://github.com/RUCAIBox/TextBox.

Although pre-trained language models (PLMs) have shown impressive performance by text-only self-supervised training, they are found lack of visual semantics or commonsense, e.g., sizes, shapes, and colors of commonplace objects. Existing solutions often rely on explicit images for visual knowledge augmentation (requiring time-consuming retrieval or generation), and they also conduct the augmentation for the whole input text, without considering whether it is actually needed in specific inputs or tasks. To address these issues, we propose a novel visually-augmented fine-tuning approach that can be generally applied to various PLMs or NLP tasks, without using any retrieved or generated images, namely VAWI. Specifically, we first identify the visually-hungry words (VH-words) from input text via a token selector, where three different methods have been proposed, including syntax-, attention- and learning-based strategies. Then, we adopt a fixed CLIP text encoder to generate the visually-augmented representations of these VH-words. As it has been pre-trained by vision-language alignment task on the large-scale corpus, it is capable of injecting visual semantics into the aligned text representations. Finally, the visually-augmented features will be fused and transformed into the pre-designed visual prompts based on VH-words, which can be inserted into PLMs to enrich the visual semantics in word representations. We conduct extensive experiments on ten NLP tasks, i.e., GLUE benchmark, CommonsenseQA, CommonGen, and SNLI-VE. Experimental results show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and outperform several competitive baselines significantly. Our codes and data are publicly available at~\url{https://github.com/RUCAIBox/VAWI}.

Dense retrieval aims to map queries and passages into low-dimensional vector space for efficient similarity measuring, showing promising effectiveness in various large-scale retrieval tasks. Since most existing methods commonly adopt pre-trained Transformers (e.g. BERT) for parameter initialization, some work focuses on proposing new pre-training tasks for compressing the useful semantic information from passages into dense vectors, achieving remarkable performances. However, it is still challenging to effectively capture the rich semantic information and relations about passages into the dense vectors via one single particular pre-training task. In this work, we propose a multi-task pre-trained model, MASTER, that unifies and integrates multiple pre-training tasks with different learning objectives under the bottlenecked masked autoencoder architecture. Concretely, MASTER utilizes a multi-decoder architecture to integrate three types of pre-training tasks: corrupted passages recovering, related passage recovering and PLMs outputs recovering. By incorporating a shared deep encoder, we construct a representation bottleneck in our architecture, compressing the abundant semantic information across tasks into dense vectors. The first two types of tasks concentrate on capturing the semantic information of passages and relationships among them within the pre-training corpus. The third one can capture the knowledge beyond the corpus from external PLMs (e.g. GPT-2). Extensive experiments on several large-scale passage retrieval datasets have shown that our approach outperforms the previous state-of-the-art dense retrieval methods. Our code and data are publicly released in https://github.com/microsoft/SimXNS

Multi-hop Question Answering over Knowledge Graph~(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question on a large-scale Knowledge Graph (KG). To cope with the vast search space, existing work usually adopts a two-stage approach: it firstly retrieves a relatively small subgraph related to the question and then performs the reasoning on the subgraph to accurately find the answer entities. Although these two stages are highly related, previous work employs very different technical solutions for developing the retrieval and reasoning models, neglecting their relatedness in task essence. In this paper, we propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning. For model architecture, UniKGQA consists of a semantic matching module based on a pre-trained language model~(PLM) for question-relation semantic matching, and a matching information propagation module to propagate the matching information along the edges on KGs. For parameter learning, we design a shared pre-training task based on question-relation matching for both retrieval and reasoning models, and then propose retrieval- and reasoning-oriented fine-tuning strategies. Compared with previous studies, our approach is more unified, tightly relating the retrieval and reasoning stages. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our method on the multi-hop KGQA task. Our codes and data are publicly available at https://github.com/RUCAIBox/UniKGQA.

With the growth of high-dimensional sparse data in web-scale recommender systems, the computational cost to learn high-order feature interaction in CTR prediction task largely increases, which limits the use of high-order interaction models in real industrial applications. Some recent knowledge distillation based methods transfer knowledge from complex teacher models to shallow student models for accelerating the online model inference. However, they suffer from the degradation of model accuracy in knowledge distillation process. It is challenging to balance the efficiency and effectiveness of the shallow student models. To address this problem, we propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. The proposed lightweight student model DAGFM can learn arbitrary explicit feature interactions from teacher networks, which achieves approximately lossless performance and is proved by a dynamic programming algorithm. Besides, an improved general model KD-DAGFM+ is shown to be effective in distilling both explicit and implicit feature interactions from any complex teacher model. Extensive experiments are conducted on four real-world datasets, including a large-scale industrial dataset from WeChat platform with billions of feature dimensions. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments, showing the superiority of DAGFM to deal with the industrial scale data in CTR prediction task. Our implementation code is available at: https://github.com/RUCAIBox/DAGFM.

预训练的语言模型（PLM）在自然语言生成（NLG）任务中取得了显着的成功。到目前为止，大多数PLM都使用大型一般语料库以无监督的方式进行了预培训。同时，与无监督的模型相比，预先训练的模型越来越多地显示出较低的数据表现出色。受监督预训练的成功的激励，我们提出了自然语言生成的多任务监督预训练（MVP）。为了预先培训文本生成模型MVP，我们从七个生成任务中收集了45个数据集的标记预训练语料库。对于每个任务，我们进一步预先训练特定的软提示，以刺激执行特定任务的模型能力。广泛的实验证明了我们在许多NLG任务中有监督的预训练的有效性，并且我们的一般方法在17个数据集中的12个中实现了最先进的性能。

会话推荐系统（CRS）旨在主动引起用户偏好，并通过自然语言对话推荐高质量的项目。通常，CRS由建议模块组成，以预测用户的首选项目和对话模块，以生成适当的响应。要开发有效的CR，必须无缝整合两个模块。现有作品要么设计语义一致性策略，要么共享两个模块之间的知识资源和表示。但是，这些方法仍然依靠不同的体系结构或技术来开发两个模块，因此很难进行有效的模块集成。为了解决这个问题，我们根据知识增强的及时学习提出了一个名为UNICRS的统一CRS模型。我们的方法将建议和对话子任务统一到及时学习范式中，并根据固定的预训练的语言模型（PLM）利用知识增强的提示来以统一的方法来实现两个子任务。在及时的设计中，我们包括融合的知识表示，特定于任务的软令牌和对话环境，它们可以提供足够的上下文信息以适应CRS任务的PLM。此外，对于建议子任务，我们还将生成的响应模板作为提示的重要组成部分结合起来，以增强两个子任务之间的信息交互。对两个公共CRS数据集进行的广泛实验证明了我们方法的有效性。

本文旨在通过介绍第一个中国数学预训练的语言模型〜（PLM）来提高机器的数学智能，以有效理解和表示数学问题。与其他标准NLP任务不同，数学文本很难理解，因为它们在问题陈述中涉及数学术语，符号和公式。通常，它需要复杂的数学逻辑和背景知识来解决数学问题。考虑到数学文本的复杂性质，我们设计了一种新的课程预培训方法，用于改善由基本和高级课程组成的数学PLM的学习。特别是，我们首先根据位置偏见的掩盖策略执行令牌级预训练，然后设计基于逻辑的预训练任务，旨在分别恢复改组的句子和公式。最后，我们介绍了一项更加困难的预训练任务，该任务强制执行PLM以检测和纠正其生成的解决方案中的错误。我们对离线评估（包括九个与数学相关的任务）和在线$ A/B $测试进行了广泛的实验。实验结果证明了与许多竞争基线相比，我们的方法的有效性。我们的代码可在：\ textColor {blue} {\ url {https://github.com/rucaibox/jiuzhang}}}中获得。

为了开发有效的顺序推荐人，提出了一系列序列表示学习（SRL）方法来模拟历史用户行为。大多数现有的SRL方法都依赖于开发序列模型以更好地捕获用户偏好的明确项目ID。尽管在某种程度上有效，但由于通过明确建模项目ID的限制，这些方法很难转移到新的建议方案。为了解决这个问题，我们提出了一种新颖的通用序列表示方法，名为UNISREC。提出的方法利用项目的文本在不同的建议方案中学习可转移表示形式。为了学习通用项目表示形式，我们设计了一个基于参数美白和Experts的混合物增强的适配器的轻巧项目编码体系结构。为了学习通用序列表示，我们通过抽样多域负面因素介绍了两个对比的预训练任务。借助预训练的通用序列表示模型，我们的方法可以在电感或跨传导设置下以参数有效的方式有效地传输到新的推荐域或平台。在现实世界数据集上进行的广泛实验证明了该方法的有效性。尤其是，我们的方法还导致跨平台环境中的性能提高，显示了所提出的通用SRL方法的强可传递性。代码和预培训模型可在以下网址获得：https：//github.com/rucaibox/unisrec。

最近，Experts（简称为MOE）体系结构在提高大规模语言模型的模型能力方面取得了巨大的成功。但是，MOE需要比要扩展的基本模型要合并更多的参数。在本文中，我们建议通过跨专家共享信息来构建一个有效的MOE架构。我们采用矩阵产品运营商（MPO，量子多体物理学的张量分解）来重建专家层中的参数矩阵，并通过共享中央张量的参数（包含核心信息）来增加预训练语言模型的模型容量（ ）在不同专家的同时，通过不同专家的辅助张量（补充中央张量）实现特异性。为了解决不平衡的优化问题，我们进一步设计了基于MPO的MOE体系结构的梯度面膜策略。基于T5和GPT-2的广泛实验表明，预训练的语言模型的性能和效率提高（与开关变压器相比，高级模型性能的总参数降低了27.2倍）。我们的代码可在\ url {https://github.com/rucaibox/mpo/mpoe}上公开获得。